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ABSTRACT 

The Legislative Office of Education Oversight (LOEO) assessed 
the planning and implementation of Ohio's Education Management Information 
System (EMIS) . The EMIS was mandated in 1989 as a provision of one of the 
most comprehensive educational reform bills ever passed in Ohio. The EMIS was 
developed based on an existing computer network, the Ohio Education Computer 
Network, which consists of the Ohio Department of Education (ODE) , regional 
computer centers (A-sites), and school districts. A-sites are the regional 
data collection centers through which school districts send their EMIS data. 
Interviews with personnel of 14 school district, A-sites, and other 
interested parties conducted by the LOEO indicate that there has been poor 
communication between the ODE and the legislature, something that has 
resulted in underestimating the size and complexity of the EMIS and 
establishing an unrealistic time frame for planning and implementing the 
system. Systems development experts would consider the EMIS highly complex, 
but state legislation allowed less than 2 years for the development of the 
EMIS, and only 23 staff members were allocated to the project. ODE has not 
provided top level leadership for the EMIS. To date, the EMIS has resulted in 
the collection of more uniform data across schools and districts as well as 
more rapid access to information about education in Ohio. Many improvements 
in the EMIS have taken place, but much remains to be done before it becomes 
fully operational. Four appendixes contain a news release, details on the 
information management systems of some other states and a history of school 
data collection. (SLD) 
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The Legislative Office of Education Oversight (LOEO) serves as staff to the 
Legislative Committee on Education Oversight Created by the General Assembly in 
1989, the Office evaluates education-related activities funded wholly or in part by the 
state of Ohio. 

This report from the LOEO to the Legislative Committee on Education Oversight 
assesses the planning and implementation of Ohio's Education Management Information 
System. Conclusions and recommendations in this report are those of the LOEO staff 
and do not necessarily reflect the views of the Committee or its members. 



SUMMARY 



AN ASSESSMENT OF OHIO'S 
EDUCATION MANAGEMENT INFORMATION SYSTEM 



The Legislative Office of Education Oversight (LOEO) assessed the planning and 
implementation of Ohio's Education Management Information System (EMIS). The EMIS 
was mandated in 1989 as a provision of one of the most comprehensive educational 
reform bills ever passed in Ohio, Senate Bill 140. 

BACKGROUND 

Senate Bill 140 created the EMIS to improve accountability by providing more 
information to policy makers and the public about school districts and schools. This act 
required the Ohio Department of Education (ODE) to collect student, staff, and financial 
data to facilitate comparisons among districts and schools. To answer legislators' 
ongoing questions about schools, ODE tried to develop a system that was flexible and 
comprehensive. 

ODE determined that the most effective way to accomplish the purposes for the 
EMIS was to create a computerized data base. ODE intended to use the EMIS as its 
primary data-collection tool and thereby replace some of its paper forms. As a result, 
the EMIS collects the new data required by S.B. 140, as well as much of the information 
ODE previously collected on paper. The 1993-1994 school year marks the third year 
school districts are submitting EMIS data. 

The General Assembly appropriated $3.6 million for the EMIS for fiscal year 1990, 
the year ODE began developing the EMIS. The fiscal year 1994 appropriation is $8.9 
million-nearly two and a half times greater than the first year's funding. Thus, although 
the initial appropriation for school districts was viewed as inadequate, this amount has 
been substantially increased in subsequent budgets. 

The EMIS was developed based upon an existing computer network, the Ohio 
Education Computer Network (OECN), established in 1979. The OECN consists of three 
parts: ODE, regional computer centers called "A-sites," and school districts. 

A-sites serve as the regional data collection centers through which school districts 
must send their EMIS data. Because the A-sites aggregate the individual student data, 
the EMIS structure assures that no personally identifiable student data reach ODE, 



FINDINGS 



Based on LOEO interviews, during legislative deliberations on S.B. 140, there was 
poor communication between ODE and the legislature. This poor communication 
resulted in the legislature underestimating the size and complexity of the EMIS and 
establishing an unrealistic time frame for planning and implementing the system. 

System-development experts would regard Ohio's EMIS as highly complex. A 
system this large and complex typically requires from three to five years and between 
50 and 100 staff to develop and implement Senate Bill 140 provided ODE less than two 
years to complete the EMIS. ODE allocated only 23 staff (eight of whom were part-time) 
to this effort. 

Ohio's EMIS is large and complex because it has over one million lines of 
computer programming code; 747 districts using the system; and three types of data 
(student, staff, and financial) that are linked to facilitate the required comparisons among 
schools. 

Underestimating the size and complexity of the system and establishing an 
unrealistic time frame had reverberating effects throughout the process to develop and 
implement the EMIS. 

Development of the EMIS 

ODE did not provide top-level leadership for the EMIS. As a result, there was 
a lack of communication and coordination across ODE divisions, which lead to confusion 
among school districts and A-sites about how to implement the EMIS. Furthermore, 
ODE did not conduct a feasibility study to assess the hardware, software, personnel, and 
training needs of local school districts. 

The short time line prevented ODE from piloting the EMIS before it was 
implemented statewide. As a result, all Ohio districts had to participate in correcting 
the initial problems that are expected of any new computer system. In addition, because 
of the legislative mandate to link the student, staff, and financial data, ODE had to 
require that all three types of data be collected at once, rather than be phased in. 

Unlike Ohio, other states LOEO examined either phased in the data to be collected 
or pilot tested the system on several districts before introducing the system statewide. 
In addition, these states did not have the legislative requirement to link student, staff, 
and financial data. 

Once the data submission began (during the 1991-1992 school year), a number of 
software problems occurred. Due to concerns about local control and the use of private 
vendors for the EMIS, ODE allowed school districts to use different software to submit 
the EMIS data. However, many districts had difficulties making their locally or vendor- 
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developed software work according to EMIS specifications. In addition, the first 
software used to aggregate the data at A-sites had some problems and had to be 
reprogrammed. 

In order for any management information system to be successful, its users must 
understand its purpose and be skilled in its use. More and better training and training 
materials were needed for personnel at ODE, A-sites, and school districts, prior to and 
during EMIS implementation. 

Current status of the EMIS 



State-level policy makers, school districts, and the general public will experience 
some immediate benefits from the EMIS. The EMIS has resulted in the collection of 
more uniform data across schools and districts as well as more rapid access to 
information about education in Ohio. 

As school districts begin the third year of submitting data, ODE is still working 
to make the EMIS fully operational. School district personnel interviewed by LOEO 
have expressed a number of ongoing concerns. Local educators are still frustrated by 
the amount of time and money required to implement the EMIS. 

In addition, many educators and their communities still do not fully understand 
the purpose of the EMIS, nor its benefits at the local level. These benefits could be 
realized if districts had software enabling them to extract EMIS information for local 
decision making. 

Educators also are concerned about having information on individual students 
included in a statewide data base. Currently, this issue is addressed by having the A- 
sites aggregate the individual data before they are sent to ODE. However, ODE needs 
access to individual student data in order to generate EMIS reports and answers to 
legislators' questions in the quickest manner. 

Technology exists that would allow the collection of individual student data 
without students being personally identified. For example, Texas assigns an arbitrary 
number to each student to be used as an anonymous identifier. 

Although ODE must address these concerns before the EMIS becomes part of 
schools' operating routines, many improvements have already taken place. Many of the 
difficulties with the EMIS occurred during the first year of data submission; ODE has 
since put into place a number of mechanisms to improve the system. Virtually everyone 
surveyed by LOEO said they think the EMIS will run more smoothly this year. 
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LOEO RECOMMENDS: 



As leader of elementary and secondary education in Ohio, ODE take 
all possible steps to continue to improve the education community's 
communication with the legislature. Such steps could include frequent 
updates on various educational issues, and dear explanations of ODE 
policy positions. 

The General Assembly allow ODE, as the EMIS expert, to continue to 
implement and maintain the EMIS as ODE believes is necessary, with 
appropriate oversight by, and accountability to, the legislature. The 
General Assembly could, in turn, continue to convey to ODE the 
information it needs from the EMIS. 

ODE continue to monitor the quality, accuracy, and validity of the 
collected data. Furthermore, the General Assembly and public should 
be aware of the limitations of the EMIS data to make comparisons and 
judgments about schools. 

ODE develop a means to collect individual student data at the state 
level without personally identifying students. 

ODE provide ongoing training to state and district users of the EMIS. 
The training policy should include strategies for training new 
employees at die state and district levels. It should also include a 
focus on how the data in the EMIS can be useful for school 
management and decision making. 

ODE develop a user manual to accompany the EMIS Guidelines. This 
user manual should be presented in a very elementary fashion for 
users who have limited experience with computers. It could be 
updated periodically in response to system or technology changes. 

ODE establish a procedure for ongoing evaluation and updating of the 
EMIS with user feedback as the primary source for possible changes. 
This should include ongoing evaluation of the EMIS codes to ensure 
accurate comparisons of schools and districts. 
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AN ASSESSMENT OF OHIO'S 
EDUCATION MANAGEMENT INFORMATION SYSTEM 



CHAPTER I 
INTRODUCTION 

In June of 1989, the 118th General Assembly enacted Amended Substitute Senate 
Bill 140 (S.B. 140), one of the most comprehensive educational reform bills ever passed 
in Ohio. Senate Bill 140 contained 39 separate provisions to be carried out by the Ohio 
Department of Education (ODE). 

One provision of S.B. 140 required ODE to develop and implement a compre- 
hensive Education Management Information System (EMIS) for elementary and secondary 
schools. According to ODE, the general purpose of the EMIS was to increase 
accountability for tax dollars and educational outcomes by school districts to state-level 
policy makers and the public. The language of S.B. 140 reflects a number of specific 
purposes for the EMIS: 

1. To obtain uniform (or standardized) data regarding various input and output 
measures of schooling. 

2. To tie operating costs to output measures to obtain efficiency ratings. 

3. To compare schools and school districts. 

4. To provide numerical data separated by school, grade level, and subject area 
to respond to different provisions in S.B. 140, including identification of 
excellent and deficient schools and districts. 

5. To establish a flexible data base for answering a variety of complex questions 
about public school practices in Ohio. 

ODE officials determined that the most effective way to accomplish the specific 
purposes for the EMIS would be to create a computerized data base system consisting of 
student, staff, and financial data. A data base is a collection of information in a computer, 
organized so it can be sorted and retrieved rapidly for various uses. The EMIS is 
structured to electronically link school districts to ODE through regional data collection 
centers called "A-sites." 

ODE also wanted to use the EMIS as its primary data-collection tool and thereby 
replace most of its paper forms. Thus, the EMIS collects the new data required by S.B. 
140 as well as some of the data ODE already collected. 
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EMIS statutory time lines 



Senate Bill 140 (effective October 1989) required ODE to plan, design, and 
implement the EMIS within 21 months— by July 1991, when school districts were required 
to begin submitting data. It was assumed that during the fall of the following school year 
(1992-1993), ODE could use the EMIS data to provide the required profile reports on 
school districts and schools. ODE did not meet this expectation. 

In a November 24, 1992 news release, ODE outlined 11 design and implementation 
deficiencies with the EMIS. These deficiencies prevented ODE from meeting the 1992 
reporting deadline. ODE's news release is provided in Appendix A. 

Summary of concerns with the EMIS 

Some members of the General Assembly believe that the EMIS could have been 
implemented much more quickly and with fewer difficulties if ODE had hired a private 
contractor to oversee the entire project. 

However, ODE personnel believe that many of the difficulties with the EMIS 
resulted from both the department's and the legislature's underestimating its complexify 
and the ambitious 21 -month time frame specified in S.B. 140 for planning and 
implementing the system. 

School district personnel throughout the state voiced concerns about the EMIS, 
particularly during the first year of data submission, the 1991-1992 school year. Educators 
said that they spent an enormous amount of time and money implementing the EMIS and 
some do not see how the system is useful for school districts. 

Many school district personnel believe the EMIS should have been pilot tested or 
phased in, rather than implemented completely in all school districts at once. In addition, 
school district personnel believe the difficulties with the EMIS resulted from ODE not 
having a complete design for the system prior to implementation. 

State funding for the EMIS 

Since the passage of the EMIS mandate in 1989, the General Assembly has 
provided various levels of funding to plan and implement the system. Exhibit 1 lists the 
funding levels for ODE, A-sites, and school districts. The funding for school districts is 
also expressed in terms of average dollars per pupil. Smaller districts received more per 
pupil than larger districts. 



EXHIBIT 1 

STATE FUNDING FOR THE EMIS 
($ in millions) 



Fiscal Year 


To ODE 


To A-sites 


To School Districts 


Total State 
Funds 








i 

: 

i 

j 


Average 
Dollars Per 
Pupil 




1990 


$0.6 


$1.2 


$1.8 


$1.00 


$ 3.6 


1991 


$0.5 


$1.2 


$1.8 


$ 1.00 


$3.5 


1992 


$0.9 


$1.6 


$3.6 


$2.00 

i 


$6.1 


1993 


$0.5 


$2.0 


$4.5 


j $ 250 


$7.0 


1994 


$0.8 


$2.5 


$5.6 


| $3.00 


$8.9 


1995 


$0.7 


$2.6 


$5.6 


j $ 3.00 


$8.9 



Other states 



Legislators and other policy makers in Ohio expressed a need for quick access to 
accurate information about elementary and secondary schools. Policy makers in other 
states have expressed similar needs. 

According to a nationwide survey by the National Education Goals Panel, policy 
makers identified the need to increase the accountability of schools as another reason for 
developing an education management information system. Individuals surveyed hope 
the systems will help schools run more efficiently. In addition to Ohio, seven states now 
have some type of a management information system for primary and secondary 
education, and 29 states are in the process of implementing systems. 
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SCOPE AND METHODS 



To assess the issues surrounding the EMIS, LOEO addressed the following 
questions: 

1. What concerns do school districts and A=sites have with the EMIS? How 
can they be resolved? 

2. What process did ODE use in planning and implementing the EMIS? How 
did the process, cost, and issues faced by ODE compare with those in the 
professional literature, in other states, and by private contractors? 

Methods 



LOEO conducted this study by reviewing literature on information system 
planning and implementation, including reports specific to Ohio and other states. 

LOEO staff also interviewed employees of 14 representative school districts and A- 
sites, ODE staff, a state legislator, employees of a computer management consulting firm, 
and state employees in Ohio and two other states who have implemented education 
management information systems. 

LOEO appreciates the assistance of the Ohio Department of Education, the 14 
school districts and A-sites, and others in the preparation of this study. As is our policy, 
they are not identified by name in this report. 

REPORT ORGANIZATION 

Chapter II describes the structure of the EMIS, including a discussion of the size 
and complexity of the system. Chapter III discusses the current operating status of the 
system, its potential benefits, and the ongoing concerns with the system. Chapter IV 
describes the history of planning and designing the EMIS, and Chapter V describes the 
history of implementing the system. Chapter VI provides conclusions and 
recommendations. 



CHAPTER II 

DESCRIPTION OF THE EMIS 



The structure and size of an information system dictates the necessary design and 
implementation procedures. This chapter describes these elements for Ohio's EMIS. 

Structure of Ohio's EMIS 

The EMIS was developed based upon an existing computer network, the Ohio 
Education Computer Network (OECN). In 1979, the 113th General Assembly mandated 
the establishment of the network. The OECN consists of three levels of organization: 
ODE, regional computer centers called A-sites, and school districts. 

The 25 self-governing A-sites are the basis of the OECN. A-sites provide computer 
services to city, local, exempted village, and joint vocational school districts, and county 
boards. A-sites are usually located in county boards of education or joint vocational 
school districts. The governing boards of A-sites are composed of representatives from 
each of the school districts they serve. 

A-sites share computation power, specialized software, data, and technical expertise 
with individual school districts called "C-sites." These C-sites are electronically connected 
to A-sites. Currently, there are 664 C-sites. A-sites charge school districts fees for the 
services they provide. 

There are school districts that are not a part of this network, which include "B- 
sites," "independent sites," and "large city school districts." These districts are referred to 
as "independent" districts in this report. Independent districts have self-contained 
computer systems and were neither served by, nor electronically connected to, the A-sites 
of the OECN. Currently, there are 83 independent districts. 

The A-sites, known as "data acquisition sites" in the EMIS, serve as the regional 
sites for data collection, validation, and aggregation through which all school districts 
must now send their EMIS data, including the independent districts. The one exception 
is the Cleveland City School District, which decided to aggregate its own data. In effect, 
A-sites serve as the ’backbone" of the EMIS. The services previously provided to school 
districts under the OECN are still available to member districts. 

The EMIS structure assures that no personally identifiable student data reach ODE. 
A-sites aggregate the student data, and send the aggregates to ODE along with staff and 
financial data. Exhibit 2 displays the structure of the EMIS. 
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EXHIBIT 2 
EMIS STRUCTURE 



DISTRICT LEVEL 



REGIONAL LEVEL 



V 



1 A- sites aggregate student data and send student, 

stalT, and financial data to ODE 



STATE LEVEL 



iz 



C -sites input student, staff, and 
financial data 


Independent sites input student, 
staff, and financial data 











Cleveland City School 
District inputs student, 
staff, and financial data; 
aggregates student data; 
and sends all data to 
ODE 



Ohio Department of Education stores and analyzes data and creates statewide, district, 
and school reports 






Note; Shaded areas represent the original Ohio Education Computer Network (OECN) 
System size and complexity 

Several factors contribute to the description of an information system's size and 
complexity: the number of lines of computer programming code; the number of different 
types of data in the system; whether these data are linked together; and the number of 
users of the system. 

Computer programmers usually refer to the number of lines of programming code 
when describing the size of an information system. Exhibit 3 gives an indication of the 
level of complexity that should be expected for variously sized systems. 



EXHIBIT 3 

SIZE AND COMPLEXITY INDEX 



Size of Systems 
(lines of code) 


Level of Complexity | 


Up to 1,000 


Trivial 1 


1,000 to 10,000 


Simple | 


10,001 to 100,000 


Difficult | 


100,001 to 1,000,000 


Complex 


1,000,001 to 10,000,000 


Nearly impossible (Ohio) 


More than 10,000,000 


Utterly absurd | 



Source: Managing: the System Life Cycle 



Considering the state-level and the A-site software together, the EMIS is a "nearly 
impossible" system, consisting of over 1,100,000 lines of programming code. The state- 
level and A-site software are "complex" systems with over 400,000 and 700,000 lines of 
programming code, respectively. 

According to the book. Managing the System Life Cycle, "complex" systems usually 
require 50 to 100 staff or more to develop and implement, including the various levels of 
managers, computer programmers, and other employees. This book also estimates that 
an information system of this complexity takes three to five years or more to complete. 
The book did not give time or staff requirements for systems described as "nearly 
impossible." 

A total of 23 staff, eight of whom were part-time, developed the 1,100,000 lines of 
EMIS code. For the state-level software, eleven full-time and one part-time staff were 
employed to develop a system with over 400,000 lines of code. For the software needed 
at tile A-sites, 11 staff were employed to develop over 700,000 lines of code. Seven of 
these worked part time. 

Because of the EMIS legislation, more information on students, staff, and school 
finances is now reported to the state than in the past. School districts now report up to 
80 data elements on each of Ohio's nearly two million students, 56 data elements on each 
staff person, and 94 financial data elements. Exhibit 4 provides examples of the 
information collected for each type of data. 
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EXHIBIT 4 

TYPES OF EMIS DATA 



STUDENT DATA 


1 

STAFF DATA 


FINANCIAL DATA 


Demographics 


Demographics 


Cost of each instructional 
program 


Attendance 


Attendance 


Cost of each 
extracurricular program 


Regular education courses 


Courses taught 


Cost of noninstructional 
activities 


Special education courses 


Extracurricular 

assignments 


Administrative cost for 8 

district 


Vocational education 
courses 


Educational degree 


Administrative cost for 
each building 


Report card grades 


Total years' experience 


Facilities acquisition & 
construction 


Test scores 


Scheduled workdays 


Debt service 


Extracurricular activities 


:: 


lii-iiiiiiM 


' 


Other uses of funds j 



Source: Ohio Department of Education 



As an example, to fulfill the legislative requirement to report on a student's 
performance, districts must provide test scores and course grades. To fulfill the 
requirement regarding student participation, districts must report each class the student 
takes (e.g., physics, vocational education) and each student' s extracurricular activities (e.g., 
drama, football). 

In order to meet the time lines of the EMIS mandate, the collection of all three 
types of data were initiated simultaneously. In addition, to fulfill the purposes of the 
EMIS mandate, it was necessary to design the system so that all three types of data would 
be linked. 

ODE consulted with Florida and Texas, which were developing their information 
systems at about the same time. These states, however, were not required to include all 
three data types (student, staff, and financial), nor to link the data types together in the 
system. After six years, Florida is just beginning to incorporate the financial data into its 
information system, and Texas has yet to include the curriculum component that ties 
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together its staff and student data. Appendix B contrasts the development of the systems 
in Florida, Texas, and Ohio. 

The complexity of a system can also be described by the number of different users 
of the system, such as school districts. Exhibit 5 shows the number of school districts for 
10 states. 



EXHIBIT 5 

NUMBER OF SCHOOL DISTRICTS BY STATE 



STATES 


NUMBER OF DISTRICTS 


Texas 


1,100 


Ohio 


747 


Minnesota 


440 II 


Wisconsin 


430 1 


South Carolina 


91 | 


Florida 


67 


West Virginia 


55 


Maryland 


24 


Delaware 


19 


Hawaii 


1 1 



Source: National Education Goals Panel 



Ohio's EMIS must serve 747 school districts. This figure includes city, county, 
exempted village, and joint vocational school districts, and county boards of education. 
Florida serves only 67 school districts. By comparison, the task of organizing 67 school 
districts in the planning and implementation of an information system is simpler than 
organizing 747. 

In short, Ohio's EMIS is a large and highly complex system. Typically, a system 
of this size and complexity would require at least three to five years to complete and 50 
to 100 staff to develop. Senate Bill 140 provided less than two years to plan and 
implement the EMIS and only 23 staff were allocated to this effort 
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CHAPTER III 

CURRENT STATUS, BENEFITS, AND CONCERNS OF THE EMIS 

This chapter describes the current operating status of the EMIS, its benefits, and 
the ongoing concerns that districts have with the system. Districts' ongoing concerns are 
being addressed by ODE. 

Operating status of the EMIS 

ODE is still working toward making the EMIS fully operational. Many school 
districts are submitting some data late or not at all for the various reporting periods. 

For each school year, school districts have six EMIS reporting deadlines. Different 
types of data (student, staff, and financial) are reported at different time periods. Exhibit 
6 shows when each type of data must be reported. 

EXHIBIT 6 

EMIS DATA-REPORTING DATES 





Student 


Staff 


Financial 8 


October 


X 


X 


x* 


December 


X* 






January 






X* 


April 






X* 


June 


X | 


1 x 




July 






X 

1 



* relatively small data submissions. 

Source: Ohio Department of Education 

As many as 124 districts (17 percent) did not report any of the required data for 
the June 30, 1993 reporting period. However, ODE said it is not unusual for numerous 
districts not to report the required data on time. In late June, many districts do not have 
personnel readily available to enter data, so districts wait until the following school year 
to report the information to ODE. In fact, 95 of the 124 districts had reported the required 
June data by September 9, 1993. 
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To address the problem of late reporting, ODE has recommended to districts to 
either begin to enter the majority of the data for the June reporting period in early May, 
or place their person in charge of the EMIS on contract for 12 months, instead of nine 
months, so the data can be entered during the summer. 

Approximately 20 districts are still having mechanical problems reporting the 
student and staff data. One district switched to a new student software package after the 
October reporting, making it impossible for it to meet the next reporting deadline. In 
another instance, a vendor for several districts has not reprogrammed the software so the 
districts can report the student data. Other districts simply are not entering the data early 
enough to meet the various reporting deadlines. 

After all school districts have submitted the EMIS data, ODE must verify its 
accuracy. Until all the data are accurate, statewide reports cannot be generated and used 
to compare schools and districts. ODE expects the first district and building profile 
reports to be published in November or December, 1993. 

Present and potential benefits 

The EMIS will immediately benefit state-level policy makers, school districts, and 
the general public. Additional benefits could be realized if there were state-level access 
to individual student data stripped of personal identifiers. Other benefits could be 
realized if districts had software enabling them to extract EMIS information for local 
decision making. 

Benefits to policy makers. When the EMIS is fully operating, policy makers will 
have quicker access to information than in the past. This information includes compari- 
sons among districts and schools. However, there are two limitations to the EMIS for 
policy makers. 

First, ODE routinely collects the appropriate EMIS information to quickly answer 
the most frequently asked questions, such as the number of students by grade, race, and 
sex. However, due to the design of the EMIS, more complicated questions will take 
longer to answer. 

For example, if a legislator requests information that is not routinely collected by 
ODE, such as how many female students tire taking advanced mathematics courses, a 
new software program must be written to gather the necessary information. Each of the 
25 A-sites must then run the program and send the results to ODE. Information from the 
25 A-sites must then be compiled to answer the question. 
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If the question is asked during the school year, when the proper school personnel 
are available to attest to the accuracy of the data, this procedure could take from 14 to 30 
days. The answer could take longer if the question is asked during the summer when 
school personnel are not readily available. 

For ODE to rapidly answer all possible questions without the assistance of A-sites 
and without requiring the development of new programming code, individual student 
data would have to be collected at the state level. 

The second constraint is that there are limits to what numbers can reveal. The 
EMIS can be an effective means of providing the number of students, staff members, and 
dollars associated with a particular school program or activity. It cannot, however, fully 
explain why a program is effective or ineffective. 

Benefits to ODE . By law, ODE has been the state agency responsible for 
monitoring and collecting information from school districts. LOEO stated in its 1990 
report. Public School Reporting Requirements, that different divisions within ODE did not 
communicate with one another and therefore collected duplicate data from districts. 

In order for ODE to successfully convert its use of paper forms to electronic EMIS 
reporting, the department had to begin to standardize the data it collects from various 
districts and improve internal departmental communications. To accomplish this, during 
the second year of data submission, ODE created an internal users' group, made up of 
personnel from each of its divisions, to review the data being collected by each division. 

The users' group improved internal communication among ODE divisions and 
increased the accuracy of the data ODE collects from school districts. As a result, the 
legitimacy of comparisons among school districts and schools has been increased. 

Benefits to districts. As a result of the EMIS, school districts also are standardizing 
their data among each other and among the schools within each district In its 1990 
report, LOEO found that school districts have to complete as many as 330 paper forms 
a year. The paper reporting burden of school districts will be reduced through the EMIS. 
To date, ODE has eliminated 26 paper forms. 

Prior to the EMIS, most districts compiled and manipulated data by hand and did 
not have rapid access to data to make management decisions. Once the EMIS is fully 
operating, districts also will have quick access to the EMIS data to use for management 
decisions. A history of state and district data collection is provided in Appendix C. 
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However, the use of the EMIS is currently limited for school districts. The 
necessary software is not available at the local level to extract the data for developing 
locally useable reports. ODE has recognized this problem and is reviewing various 
software packages to purchase for districts for this purpose. 

Benefits to the general public. When the EMIS is fully operating, the general 
public will have access to much more information about schools and districts than ever 
before. Ohio Revised Code section 3301. 0714(F) (l)-(3) requires the State Board: 

to annually prepare a statewide report that includes the profile 
of each of the districts [and] . . . annually prepare an 
individual report for each school district . . . that includes the 
profiles of each of the school buildings in that district 

Copies of the statewide and district reports are to be sent to each district and its 
board of education. These reports also are to be made available to the public. ODE was 
scheduled to have the first reports available during the fall of 1992, but as noted, 
difficulties with the system prevented them from meeting this expectation. ODE is 
currently working to finalize the format of these reports for November or December, 1993. 

Ongoing concerns with the EMIS 

All but one of the representatives of the 14 A-sites and school districts surveyed 
by LOEO believe the EMIS will run more smoothly during the 1993-1994 school year. 
However, all district respondents stated that ongoing concerns need to be addressed 
before the EMIS becomes a part of their schools' operating routine. Four of the most 
frequently mentioned ongoing concerns were: 

1. Use of educators' time implementing the EMIS. 

2. Cost of implementing the EMIS. 

3. Student privacy issues resulting from collecting student-identifiable 
information. 

4. Educators' and the public's lack of understanding of the purpose of the 
EMIS. 

Time consuming for educators. All district personnel interviewed by LOEO 
identified the time it takes to implement the EMIS as a major concern. Many districts did 
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not have the time nor the staff to devote solely to the EMIS; most districts had to add the 
EMIS to staff's existing duties. Furthermore, during the first year of data submission 
(1991-1992), school district personnel had to learn to operate the EMIS while entering the 
district' s data to meet the first reporting deadline of October 1991. 

As noted, one reason the EMIS was so time consuming is that schools have to 
report up to 80 data elements on each student However, districts also have to determine 
which code should be used for each of the 80 data elements. For example, if a student 
is enrolled in vocational education, there are 209 coding options to choose from to specify 
exactly which vocational education course the student is taking. This makes districts' task 
of inputting student data very complex and time consuming. 

Costly implementation. The 118th General Assembly gave school districts $1.8 
million for EMIS for each year of the 1989-1991 biennium. This funding provided 
approximately $1.00 per pupil per year. It was to be used for all costs associated with 
EMIS, such as purchasing hardware and software and hiring personnel. 

Based on LOEO's interviews, the amount appropriated was not enough for many 
districts. ODE observed that, "The appropriation was insufficient to support staff 
development, software development and purchase, and hardware purchase to implement 
the system." 

The cost to school districts for implementing an information system depends upon 
whether they have available staff and how computerized the districts are in the first place. 
This finding is consistent with what LOEO found in other states that have information 
systems. For example, prior to the EMIS, some school districts had only one outdated 
computer; other districts had no computers at all; while still other districts were 
completely computerized. 

Ohio school districts interviewed by LOEO report that they had to supplement 
more than twice the amount appropriated by the legislature. For example, a large-city 
school district reportedly spent nearly $500,000 more than what was appropriated. 
Statewide, district expenditures were for extra personnel, office equipment and space, and 
hardware. One district postponed the expense of hiring new staff because the secretary, 
principal, and school counselor volunteered time to work on the EMIS. 

The 119th General Assembly gave approximately $3.6 million to school districts for 
the first year of entering data into the EMIS and $4.5 million for the second year (school 
years 1991-1992 and 1992-1993). This was approximately $2.00 per pupil for the first year, 
and $2.50 for the second year. ODE determined that the funding was still not enough to 
meet school districts' EMIS expenses. 
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The 120th General Assembly agreed with ODE that districts needed more money 
to meet the EMIS mandate. The legislature responded by providing, in Amended 
Substitute House Bill 152, $5.6 million to school districts for the third and fourth years of 
submitting data. This is approximately $3.00 per pupil per year. 

Student privacy. Representatives from all the school districts interviewed by LOEO 
voiced concerns that collecting student-identifiable information violates students' and 
families' rights to privacy. 

In Amended Substitute House Bill 152, the 120th General Assembly prohibited the 
reporting under EMIS of any personally identifiable student information, including social 
security numbers, to the State Board of Education or to ODE. In addition, any person 
releasing EMIS data to legally unauthorized individuals is subject to criminal penalty for 
tampering with data. This is a fourth-degree felony. 

However, as noted, in order for ODE to generate reports from the EMIS in the 
quickest manner, individual student data must be collected at the state level. Technology 
allows the collection of individual student data without students being personally 
identified. 

For example, Texas has technology placed at regional sites (similar to Ohio's A- 
sites) which assigns a number to each student to be used as a record identifier. This 
technology then eliminates any personally identifiable information, such as the social 
security number and name, before the individual student data are sent to the state. 

Lack of understanding of the EMIS purpose. Based on LOEO interviews, repre- 
sentatives of many districts and A-sites do not fully understand the purpose of the EMIS, 
nor how it can enhance district operations. One respondent said, "Frankly, I have never 
been given a clear purpose ... I have no idea [what it is to be used for]." 

Several district representatives repeatedly stated that ODE needs to help schools 
and districts understand the purpose and use of the EMIS, particularly its value to school 
districts. One A-site said that "most local districts have been so overwhelmed by the 
amount of data that they have not thought of the next step to ask, 'What can I use the 
data for?'." 

Principals and other administrators need to learn how the information in a data 
base such as the EMIS can be useful for local decision making. In order for the local 
benefits of the EMIS to be realized, ODE must demonstrate to school administrators how 
its information can help them operate and improve their schools. 
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CHAPTER IV 

HISTORY OF PLANNING AND DESIGNING THE EMIS 



ODE was required to have the EMIS fully operational within 21 months, but 
various difficulties did not allow them to meet this expectation. The 1993-1994 school 
year marks the third year of submitting EMIS data. 

The remainder of this report describes the process ODE used to plan, design, and 
implement the EMIS. The interpretation of the events discussed in the following chapters 
are based on LOEO interviews. 

PLANNING FOR THE EMIS 

The initial planning decisions for the EMIS established the context for subsequent 
events. These decisions resulted from the General Assembly's deliberations about S.B. 140 
and ODE's actions following enactment of the law. 

The General Assembly' s deliberations 

According to LOEO's interviews, the EMIS was initiated due to legislators' 
concerns with ODE's inability to answer detailed questions about elementary and 
secondary education. In the opinion of some individuals, there was an atmosphere of 
"frustration" with the education community that characterized the entire legislative 
process leading to S.B. 140. Any attempt by ODE to influence or change the language in 
S.B. 140 would have been viewed by the legislature as obstructing the educational reform 
process. As summarized in one interview: 

There wasn't a good atmosphere between ODE and the 
legislature so no one sat down and explained what it means 
to collect data on two million students. The legislature was not 
brought up to speed on the complexity and time-consuming 
nature of this undertaking. 

The size and complexity of the EMIS also were not conveyed to school districts and 
A-site personnel. A district representative said, "We didn't know what we were getting 
into . . . No one said how monumental this would be." 

Many people at the state level did not believe S.B. 140 would pass— and conveyed 
this belief to school districts. As one respondent explained: 
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Many districts or superintendents thought that if legislators 
saw that districts didn't have the hardware or software, nor 
the money to purchase any, nor the technical expertise for 
EMIS, that the legislators would back off of EMIS, or change 
the time frame. So [districts] didn't gear up quickly enough 
after S.B. 140 was actually passed. Many of the mechanical 
problems that faced EMIS resulted from this situation. 

Because of the lack of communication between ODE and the legislature, the 
department was unable to influence the legislative process to allow for such changes as 
more time to plan and implement the EMIS or to pilot test the system. ODE's inability 
to influence legislative changes to S.B. 140 had reverberating effects throughout the 
remainder of the process for planning, designing, and implementing the EMIS. 

ODE's decisions 



ODE made three major decisions that eventually affected its development of the 
EMIS: the choice of leadership for the EMIS; not initiating a policy requiring internal 
coordination; and not conducting a feasibility study of school districts' computer needs. 
The first two issues are related and bring into question the extent to which the highest 
levels of ODE management were committed to the EMIS. 

EMIS leadership. Leadership has been identified as an essential variable for 
successfully implementing a statewide information system. A representative from Florida 
identified "effective leadership" as a key reason Florida was successful in implementing 
its system. With a high-level official leading Florida's efforts, department employees, 
school districts, and other organizations took the system seriously and cooperated. 

In Ohio, ODE initially designated a mid-level manager to lead the EMIS, while 
maintaining that individual's existing duties. Top-level management at ODE did not 
assume responsibility for the EMIS. One individual described Ohio's experience: 

What I saw happening was that ODE had a lot of different 
projects to deal with as a result of S.B. 140 and top 
management assigned these projects out. Top management 
didn't give EMIS the priority that was necessary either 
because top management didn't want to devote the time or 
didn't understand the extensiveness of the undertaking. 

Lack of internal coordination. When ODE was planning the EMIS, the department 
consisted of five divisions with over 50 programs among them. Various ODE divisions 
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did not initially cooperate with the development of the EMIS. As late as January 1993, 
over one year after the schools began submitting data, ODE still had no department-wide 
policy coordinating its divisions or requiring EMIS training for till division personnel. 
Some division directors have since made EMIS training mandatory. 

Between July and December 1989, ODE hired four external consultants to assist in 
planning for the EMIS. One consultant's assessment of ODE's internal operations 
concluded that multiple definitions of data existed within and among divisions. The 
consultant also concluded that the leadership required to solve these and other concerns 
must come at the superintendent or assistant superintendent level. 

No feasibility study. ODE did not conduct a feasibility study of districts' hardware 
and software needs and the technical capabilities of districts' personnel. A 1989 study 
completed prior to the EMIS legislative mandate concluded that one of the main obstacles 
to implementing a statewide information system was the lack of technical expertise of 
personnel at school districts. ODE officials said due to the ambitious time frame for 
completing the EMIS, they would not have had time to use such a feasibility study, even 
if one had been completed. 



DESIGNING THE EMIS 

According to system-development literature, failure to acknowledge or understand 
the importance of completing a design "always lead[s] to serious deficiencies" in the 
system later in the process. School personnel and others describe the EMIS as poorly 
designed. According to one respondent: 

EMIS was haphazardly put together. There was no design. 

ODE panicked . . . [and] began writing [code] as they went 
along . . . Why else would they be six months into the 
program and have to redesign it? Because they didn't think 
about a design before. 

ODE also acknowledges that the EMIS was poorly designed, saying, "The 
specification for developing the system lacked precise definition." 

Insufficient district involvement 

Districts had two major ways to provide input on the design of the EMIS. 
Representatives of school districts participated on a committee which initially identified 
the data to be collected. In addition, ODE held one all-day statewide meeting to further 
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discuss the EMIS data. However, most respondents felt that there was insufficient 
opportunity to provide input on the EMIS design. 

According to information system development literature, users have the most 
intimate knowledge about the data gathered and analyzed in the system and recommend 
that users work closely with system developers as "partners in development." 

No pilot testing or phasing in the system 

ODE decided not to pilot test nor to phase in the EMIS. ODE employees 
interviewed by LOEO maintain that they wanted to pilot test the EMIS but felt pressure 
from some legislators who believed that taking the time to do so would mean the system 
would never be implemented. 

In a 1990 Buckeye Association of School Administrators' survey of all district 
superintendents, 99 percent of the respondents felt that "the [E]MIS should be field-tested 
in several areas of the state." In fact, virtually everyone LOEO interviewed expressed that 
ODE should have piloted or phased in the EMIS. 

According to system-development literature, piloting or testing the system prior 
to full-scale implementation is absolutely necessary. In addition, according to a survey 
of 50 states by the National Education Goals Panel, few states bring all districts and data 
elements on line at once. 

Florida's system was pilot tested for nearly three years. Florida also did not bring 
all the data types (student, staff, and financial) into the system at once. Texas gradually 
phased in its data types over three years. The EMIS legislation, however, required that 
all data types be available simultaneously. 

EMIS software problems 

Software to run the EMIS is needed at three levels: the state (ODE); the A-sites; 
and the school districts. School districts are using a variety of different types of software 
for inputting one or more of the three groups of data. This lack of software uniformity 
created problems. 

Under the Ohio Education Computer Network, school districts had been allowed 
to use whatever software they wanted to transmit the required financial accounting data 
to ODE. Some districts used the ODE software developed for this purpose; others 
developed their own software in-house; and still others purchased it from a variety of 
commercial vendors. 
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Because controversial issues of local control and private sector involvement were 
raised during the design of the EMIS, ODE did not specify one particular software 
program to be used, nor did ODE develop software for districts to use to input the 
student data. Instead, ODE told districts how the data had to be organized and formatted 
to be transmitted to A-sites. 

ODE considered hiring an outside consultant to design and develop the 
programming code for the entire EMIS, but later determined that no consultant knew 
enough about the specifics of Ohio elementary and secondary education. Therefore, ODE 
developed the state-level software for the EMIS in-house. 

For the regional or A-site level, ODE contracted with the Northwest Ohio 
Computer Association (NWOCA) A-site to develop the A-site aggregation software. 
NWOCA, in turn, contracted with a private consulting firm to design and lead the 
development of the aggregation software, with the assistance of ODE and some A-site 
programmers. 

Independent school districts 

According to ODE and some school districts, the relationship between ODE and 
independent school districts was very negative, because these districts were not part of 
the Ohio Education Computer Network upon which the EMIS was based. While planning 
for the EMIS, ODE concluded that the independent districts would present some 
operating problems and urged them to join the network. 

In addition, during the first year of submitting data, A-sites were charging 
independent districts higher service fees than member districts were paying. ODE's and 
A-sites' practices reinforced the negative feelings between all parties and further 
exacerbated difficulties with implementation. 

During the second year of submitting data, ODE began working to repair the 
department's relationship with these districts and encouraged A-sites to charge 
independent and member districts the same fees for EMIS services. 
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CHAPTER V 

HISTORY OF IMPLEMENTING THE EMIS 



According to system-development literature, "the availability of committed and 
skillful people," is one key to successfully implementing an information system. Based 
on LOEO interviews, it appears that representatives of many A-sites and school districts 
do not fully understand the purpose of the EMIS, nor how it can enhance district 
operations. Because many districts still do not understand the purpose, nor have come 
to realize the benefits of the EMIS, they are not committed to the system or its success. 
Without effective training, districts could not acquire the necessary skills. 

Inadequate trainine 

According to A Game Plan for Systems Development, training is essential to a 
successful system: 

Ideally, training should start long before implementation .... 
Training is perhaps the single most critical activity for all parties, 
since the success of training users will ultimately determine the 
acceptability of the system and its components. Users must not only 
know how to use the system but they must believe in it, understand 
it, feel comfortable with it, and recognize it to be an enhancement of 
their own position. 

ODE acknowledges that "local, regional, and state staff at all levels lack the training 
necessary to carry out their responsibilities." Most respondents said that ODE training was 
inadequate. Those who said training was adequate also described it as 'better than 
nothing." 

ODE did not begin training users of the system until a few months before districts 
had to begin submitting data. This training consisted of informational and question and 
answer sessions about the data to be collected. The sessions were regional in scope with 
an average of 100 people attending each session. 

Respondents stated that ODE's regional meetings were not effective because they 
were too large for district personnel to ask detailed questions. District personnel wanted 
more specific help with their individual concerns, but only received general information 
about how the EMIS was going to operate. 

Conversely, the computer specialists at the A-sites needed training in how school 
programs operate in order to carry out their responsibilities. According to the A-site and 
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district respondents, during the first year of submitting data, ODE did not offer enough 
assistance to either the A-sites or the districts. 

At the state level, personnel in various ODE divisions also did not understand how 
the EMIS operated. Consequently, when school district personnel asked ODE a question 
about how to enter data, the answer varied depending on who answered the phone. 
Districts concluded from the phone calls to ODE that the department personnel knew no 
more about the EMIS than they did. Other districts concluded that ODE had not thought 
through the EMIS. 

Further exacerbating the problem for districts was ODE's new policy, beginning 
the second year of data submission, that funding for some programs depended on getting 
the EMIS data accurate. 

Inadequate trainine materials. According to a 1972 report by the Nevada 
Department of Education: 

Every precaution [must be taken] to see that user manuals, the 
database and report dictionary, and requested procedures are 
designed for the non-technical user. 

ODE provided districts and sites with over 500 pages of definitions, procedures, 
and guidelines in a manual called the EMIS Guidelines. Many district personnel 
interviewed by LOEO expressed concern that ODE's manual is confusing and difficult to 
read. Others said that the manual was helpful, but inadequate. Two respondents noted 
that the EMIS Guidelines was designed for the A-sites or for programmers, not for school- 
level users. 

Obstacles to implementation 

As noted, most districts identified the time and cost of implementation as the major 
concerns with the EMIS. One respondent described ODE's implementation of the EMIS: 

Implementation was a confusing mess because there was 
nothing written down. Student, staff, and financial data were 
submitted virtually simultaneously— within two weeks of each 
other. 

Four additional obstacles to successful implementation of the EMIS became 
apparent after the first year of submitting data: 
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1. The aggregation software needed to be reprogrammed. 

2. The error-checking procedure for the data was inadequate. 

3. School districts were having difficulties inputting student data with the variety 
of different vendor- or district-developed software. 

4. Some A-sites were not able to handle the amount of data districts transmitted 
to them; ODE labeled these sites "weak A-sites." 

These obstacles resulted in school district personnel spending a great deal of 
unnecessary time on the EM1S. 

Aggregation software. During the first year of data submission (1991-1992), ODE 
made over 24 changes to the A-sites' aggregation software. A-sites reported that just 
when they would get trained on one aggregation procedure, ODE would change it. 

ODE acknowledges that, "Software used to aggregate data at the regional level was 
not field-tested. As a result, additional errors were introduced into the system which 
resulted in added time and costs to the school districts." 

Error-checking p rocedure. Another obstacle was that during the first year of 
submitting data, ODE had a very confusing and cumbersome way of checking for errors 
in the EMIS. During the second year, ODE streamlined and improved this process by 
having the A-sites check for errors before the data were aggregated and sent to ODE. 

The process of checking for errors could have been simplified if ODE had run a 
parallel system. A parallel system involves submitting paper forms in combination with 
electronically submitted data. ODE did not initially require districts to submit paper 
forms along with the electronically transmitted data. 

One respondent said, ODE "made a decision to just jump into it, go whole hog and 
not run a parallel system ... It wasn't a good decision not to run a parallel system." 
During the second year of submitting data, ODE introduced two forms to be submitted 
in parallel with the electronically submitted financial data. 

Softwar e for inputting student data. As. noted, the multiple versions of district 
software created implementation difficulties. Many school districts were having trouble 
making their vendor- or district-developed software work with the EMIS software. 
Several respondents concluded that ODE should have developed the student data 
software for all school districts to use. ODE concluded that many of the districts' 
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problems were caused by vendors who could not reconfigure district software for the 
EMIS, as the vendors had promised. 

"Weak” A-sites. A fourth concern was that some A-sites were not able to handle 
the amount of data that school districts were transmitting. According to some ODE staff, 
these A-sites were poorly run and managed prior to the EMIS (under the Ohio Education 
Computer Network); the EMIS merely brought the problems to the surface. 

ODE acknowledged that, "Several A-sites were incapable of meeting the demands 
of EMIS." However, this problem would have been discovered if ODE had conducted a 
feasibility study. ODE is currently developing minimum operating standards for all A- 
sites. 

Concerns about use of the EMIS data 

Personnel from nearly all the school districts and A-sites believe that the primary 
purpose for the EMIS is to make comparisons among schools and districts. However, 
districts expressed three concerns about using the EMIS data for comparing districts and 
schools. First, several districts voiced concerns that there are a variety of definitions for 
the same data. For example, one term such as "intervention" can have multiple meanings. 

Second, most A-sites said that some districts are not as careful about accurately 
inputting some data as other districts. Some districts said they are very hesitant to report 
such things as drop-out data because many districts may be reporting it incorrectly. 

Third, two districts believe that the state does not collect the right type of data to 
make an accurate comparison among districts or schools. They said that there are too 
many factors, such as socioeconomic status and mobility of students across districts, that 
are not collected but must be considered when comparing achievement results. However, 
some districts have criticized ODE for collecting too much student information. 

To summarize the EMIS history, the EMIS legislation did not permit time to pilot 
test nor phase in the system. With piloting or phasing in, many of the school districts' 
concerns would have been discovered and resolved before full implementation began. 
This would have lessened the amount of unnecessary time educators spent trying to work 
with a flawed system. 
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CHAPTER VI 

CONCLUSIONS AND RECOMMENDATIONS 



LOECXs interviews yielded conflicting information about exactly what ODE 
conveyed to the legislature during deliberations for the EM1S. It is dear, however, that 
communication between ODE and the legislature was inadequate, considering the 
magnitude and complexity of a project such as the EMIS. 

Due to a number of factors, planning and implementing the EMIS have been 
difficult. As a consequence, ODE could not meet the expectation to provide profile 
reports on school districts and schools during the fall of 1992. 

A major reason for the difficulties, as one legislator noted, is that the EMIS forced 
a re-engineering of the way the state and school districts conduct business. More 
important, however, is that the EMIS is a large and highly complex computer system, 
consisting of over 1.1 million lines of computer programming code and 747 different 
users. 



System-development experts estimate that a system as large and complex as the 
EMIS typically takes from three to five years to complete, and requires from 50 to 100 
staff to plan and implement. Senate Bill 140 provided less than two years to complete the 
EMIS and ODE allocated only 23 staff for this effort Eight of these staff were part-time. 

Other states which have implemented management information systems have done 
extensive pilot testing or have phased in the data collection. The EMIS, however, was not 
pilot tested, nor were the three types of data (student, staff, and financial) phased in, 
which resulted in many implementation difficulties. 

The 1993-1994 school year marks the third year of data submission for the EMIS. 
ODE has begun to put into place a number of mechanisms to solve many of the 
difficulties which occurred mostly during the first year of submitting data. Despite 
numerous concerns with the EMIS, districts appear to be optimistic about its future. 
Virtually everyone surveyed by LOEO said they think EMIS will run more smoothly this 
year. 



However, all local personnel stated that they have ongoing concerns that need to 
be addressed before EMIS becomes a daily part of their lives. All mentioned the issue 
of student privacy as an ongoing concern. All of those interviewed identified the time 
and cost of the EMIS, the lack of personnel, and questions about how the EMIS will be 
used as ongoing concerns. 
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RECOMMENDATIONS 

The General Assembly and ODE 

In 1989, during legislative deliberations about the EMIS, the communication 
between ODE and the legislature was inadequate. 



LOEO RECOMMENDS: 

* That as leader of primary and secondary education in Ohio, ODE take all 
possible steps to continue to improve the education community's 
communication with the legislature. Such steps could include frequent 
updates on various educational issues, and dear explanations of ODE policy 
positions. 



Many of the difficulties identified in this report occurred during the first year of 
data submission. ODE has since put into place a number of mechanisms to solve them 
and improve the EMIS. Due to the progress ODE has made toward making the EMIS 
operational. 



LOEO RECOMMENDS: 

* That the General Assembly allow ODE, as the EMIS expert, to continue to 
implement and maintain the EMIS as ODE believes is necessary, with 
appropriate oversight by, and accountability to, the legislature. The General 
Assembly could, in turn, continue to convey to ODE the information it needs 
from the EMIS. 



Use of the EMIS data 

The EMIS legislation mandates that the data be used for comparison and 
accountability purposes. Many school district personnel have concerns regarding the use 
of EMIS data to compare and make judgments about schools. The data collected by the 
EMIS cannot fully explain why a program is effective or ineffective. 
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LOEO RECOMMENDS: 

* That the ODE continue to monitor the quality, accuracy, and validity of the 
collected data. Furthermore, the General Assembly and public should be 
aware of the limitations of the EMIS data to make comparisons and 
judgments about schools. 



Individual student data 



Technology permits Ohio to collect individual student data at the state level, 
without the student being personally identified. ODE has not yet made use of this 
technology. Doing so would allow ODE to respond more rapidly to legislative and other 
informational requests, while addressing the public's concerns about student and family 
privacy. 



LOEO RECOMMENDS: 






L 



That ODE develop a means to collect individual student data at the state 
level without personally identifying students. 



Training 

The training of all users of the EMIS was identified by experts as one of the most 
important aspects of system development, implementation, and ongoing operation and 
maintenance. 



LOEO RECOMMENDS: 

* That ODE provide ongoing training to state and district users of the EMIS. 
The training policy should include strategies for training new employees at 
the state and district levels. It should also include a focus on how the data 
in the EMIS can be useful for school management and decision making. 
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Training materials ' 



Local users of the EMIS are primarily professional educators, not technicians. 
Training materials are critical as current and future users in Ohio school districts attempt 
to make EMIS part of ongoing routines. The success of EMIS is determined by users' 
acceptability, use, and comfort with the system. 




LOEO RECOMMENDS: 



* That ODE develop a user manual to accompany the EMIS Guidelines. 
This user manual should be presented in a very elementary fashion for 
users who have limited experience with computers. It could be 
updated periodically in response to system or technology changes. 




Ongoing evaluation of the EMIS 

It is not clear whether ODE has a policy or process for ongoing evaluation, 
monitoring, and updating of the EMIS. Such a policy or procedure is necessary in order 
for the system to be able to respond to changing information needs. 



LOEO RECOMMENDS: 

* That ODE establish a procedure for ongoing evaluation and updating 
of the EMIS with user feedback as the primary source for possible 
changes. This should include ongoing evaluation of the EMIS codes 
to ensure accurate comparisons of schools and districts. 
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APPENDIX A 

OHIO DEPARTMENT OF EDUCATION 
NOVEMBER 25, 1992 NEWS RELEASE 



Nov. 25, 1992 



(614) 752-4826 



OBIO DEPARTMENT OF EDUCATION IDENTIFIES KEY CONCBRNS WITH 
EDUCATION MANAGEMENT INFORMATION SYSTEM ~ 
BXCBLLBNT-DBFICIBNT RATING OF SCHOOLS POSTPONED 

COLUMBUS — The Ohio Department of Education announced today 
that another year is needed to resolve problems with using data from 
the state's Education Management Information System (EMIS) to 
determine excellent and deficient schools and districts. 

Therefore, 1991-92 school year data collected through EMIS will 
not be used to classify schools and districts as excellent or 
deficient. 

Administrative Code rules require the Department to make these 
determinations in the first year after the full implementation of 
EMIS. 

"After much analysis, we have decided that too many disparities 
still exist to consider the EMIS data reliable," said Superintendent 
of Public Instruction Ted Sanders, "it would be absolutely wrong to 
rate schools on data that, at this point, we know are flawed." 

The state school chief pointed out an additional concern with 
the excellent-deficient requirement! "In my opinion, we don't have 
the definitions quite right for excellent and deficient schools and 
districts. I am not convinced that the data elements we have 
selected as the key factors for this classification truly represent 
what schools and districts should be offering as evidence of their 
excellence . " 

— MORE -- 
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"It is unreasonable to expect schools to be rated through a 
system that we at the Department don't fully believe in," Sanders 
added. "However, we are committed to making the excellent and 
deficient system work. We will continue to collect EMIS data as we 
work through our problems. Schools and districts should be held 
accountable for what they are doing to improve student learning." 

Ohio's EMIS collects data on students, staff and finance. 

Currently, the EMIS data to be used for determining excellence 
or deficiency are (1) achievement and ability test scores in grades 
4, 6, 8, and 10, (2) ninth-grade proficiency test scores, (3) 
student and staff attendance rates and (4) student dropout rates. 

"It is the Department's responsibility to take the lead in 
making EMIS and the excellent-deficient system work, and work 
correctly. The EMIS and the Ohio Education Computer Network create 
a powerful information system for schools and districts," Sanders 

said. 



Almost all of Ohio's 611 school districts have completed 
connections to the OECN, which offers services such as connections 

to the EMIS data and the following* 

* Electronic mail for immediate communication within districts, 
w ith the Department, and with other users on compatible 
systems such as CompuServe, AppleLink, INTERNET, BITNET and 

OARNET 

* State,’ local and district data, such as pupil-teacher ratios 
and per-pupil spending figures, that may affect local policy 
decisions. 

* Financial accounting programs. 

* Data on teacher certification and university accreditation. 

* Word and document processing, spreadsheets, graphics and 

statistical analyses. . . . iat 

* Electronic connections to libraries as far away as Australia 
and as subject-specific as updates from the National 
Aeronautics and Space Administration. 
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As part of its EMIS analysis. Department staff identified 11 
key problems for implementing the system, which follow. Sanders 
noted that the problems were interdependent, and that fixing one 
would not alone ensure the success of the EMIS. 



ASSESSMENT OF THE EMIS SITUATION 

1. Multiple vendor student software packages have been created that 
are not compatible. 



2. The legislation does not authorize the state to collect 
identifiable student data. As a result, the best the state system 
can do is edit aggregated data for internal consistency and return 
for verification or correction. 



3. Software used to aggregate data at the regional level was not 
field-tested. As a result, additional errors were introduced into 
the system which resulted in added time and costs to the school 
districts. 



4. The EMIS Guide is written in a way that is not usable to 
nontechnical people. As a result, school-level clerks have 
difficulty entering data and correcting errors when they are 
identified. 

5. a decision was made to expand data collection beyond that which 
was required by legislation (e.g. eliminate paper reporting; 
eliminate redundancy? collect federally-required common core data)# 

6. The specification for developing the system lacked precise 
definition and detail. The software used at the state to edit the 
aggregated data was inadequate to verify accuracy of the data. 

7. Several A-sites were incapable of meeting the demands of EMIS. 



8. The historical friction between A-sites and B-sites exacerbated 
the problem of incompatible software packages. 

9. Multiple data bases within the Ohio Department of Education 
limited the ability to implement the system. 

10. Local, regional and state staff at all levels lack the training 
necessary to carry out their responsibilities. 

11. The appropriation was insufficient to support staff development, 
software development and purchase, and hardware purchase to 
implement the system. 



-- MORE — 
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As part of its EMIS analysis, Department staff identified 11 
key problems for implementing the system, which follow. Sanders 
noted that the problems were interdependent, and that fixing one 
would not alone ensure the success of the EMIS. 



ASSESSMENT OF THE EMIS SITUATION 

1. Multiple vendor student software packages have been created that 
are not compatible. 

2. The legislation does not authorize the state to collect 
identifiable student data. As a result, the best the state system 
can do is edit aggregated data for internal consistency and return 
for verification or correction. 

3. Software used to aggregate data at the regional level was not 
field-tested. As a result, additional errors were introduced into 
the system which resulted in added time and costs to the school 
districts • 

4 . The EMIS Guide is written in a way that is not usable to 
nontechnical people. As a result, school-level clerks have 
difficulty entering data and correcting errors when they are 
identified. 

5. A decision was made to expand data collection beyond that which 
was required by legislation (e.g. eliminate paper reporting? 
eliminate redundancy; collect federally-required common core data). 

6. The specification for developing the system lacked precise 
definition and detail. The software used at the state to edit the 
aggregated data was inadequate to verify accuracy of the data. 

7. Several A-sites were incapable of meeting the demands of EMIS. 

8. The historical friction between A-sites and B-sites exacerbated 
the problem of incompatible software packages. 

9. Multiple data bases within the Ohio Department of Education 
limited the ability to implement the system. 

10. Local, regional and state staff at all levels lack the training 
necessary to carry out their responsibilities. 

11. The appropriation was insufficient to support staff development 
software development and purchase, and hardware purchase to 
implement the system. 

— MORE — 
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With Sanders's lead/ Department staff also identified the next 
steps that must be taken to resolve the key problems. 



ACTION STEPS 

1. Identify the weak A-sites. Give appropriate period of time to 
correct deficiencies. For those who cannot/ suspend license and 
funding/ and reassign districts to other A-sites. 

2. Require vendor software to meet guidelines of required EM IS 
reporting. Those vendors whose software meets such guidelines will 
be on an approved list. This list will be made available to school 
districts wishing to use private software vendors. 

3. The Ohio Department of Education will engage a qualified 
consultant to study and make recommendations regarding the current 
organization of the Division of computer Services and Statistical 
Reports . 

4. identify and eliminate superfluous data collection. 

5. The Ohio Department of Education will expand internal and 
external staff training development related to the implementation of 

the EMIS. 

6. The Ohio Department of Education will review and revise A-site 
standards to assure that an appropriate level of current and future 
services are provided to all districts. 

7. The Ohio Department of Education will revise the standards to 
allow districts to purchase services from any A-site on a 
competitive basis. 

8. Equalize payments for servicing B-sites and C-sites. 

9. Seek appropriations that are adequate to implement the system. 

10. Because EMIS has not been fully implemented/ the Department will 
not classify schools and districts as excellent or deficient this 
year. 



— 30 — 



For more information, call Rebecca Stevens at (614) 752-4826 or 
Andy Qualtire at (614) 752-8731. 
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APPENDIX B 

COMPARISON OF OHIO'S, FLORIDA'S, AND TEXAS' 
INFORMATION SYSTEMS 



Ohio drew upon the experiences of Florida and Texas when designing and 
implementing the Ohio Education Management Information System (EMIS). The three 
states' experiences are compared below. 

Origins of the system. All three states' information systems originated with a 
legislative mandate, although Florida's system was proposed by its department of 
education. 

Time frame for completing the system. Ohio was given one year and nine months 
to have its system fully operational, while Florida spent six years to make two-thirds of 
its system operational, and Texas spent three years to develop and three years to refine 
its system. 

Pilot testing and types of data. Neither Ohio nor Texas pilot tested their systems, 
but Florida spent three years pilot testing its system. 

Ohio requires that three types of data (student, staff, and financial) be submitted 
to the department, and that the data be linked to each other. Florida initially required 
two types of data, student and staff. After spending six years to refine those elements, 
Florida is just now beginning to collect financial data, and is not required to link its 
types of data. 

Texas had automated its data collection from schools prior to the mandate to 
develop a management information system. Texas spent three years adding one type 
of data per year to implement the new system. A curriculum component, which will 
link students and staff, will be the next data added to the system. 

System size and complexity. There is little comparable information available 
regarding the complexity of these states' systems. However, one comparable 
characteristic of the individual systems is the number of school districts that are part of 
the three states' systems: Ohio has 747 districts; Texas, 1,100; and Florida, 67. 

Built on existing systems. Similar to Ohio, Florida's information system was built 
upon an existing system, and has regional sites similar to Ohio's A-sites. Texas school 
districts had automated financial, staff, and student data prior to the information system 
mandate. These three systems were connected to form the statewide system. 
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Relationship with legislature. Individuals' from both Florida and Texas report that 
their departments of education had positive relationships with their legislatures, which 
was a key factor in system implementation. The legislatures in these two states allowed 
their respective departments of education to use their own approach to implementation. 

However, the relationship between the Ohio Department of Education and the 
General Assembly was strained during EMIS deliberations in 1989, and the legislature 
gave some specific directions on the time frame and the process for implementing the 
EMIS. 



Leadership for implementing the system. Individuals from Florida and Texas 
reported that each state's system had strong leadership from individuals in positions 
equivalent to Ohio's Superintendent of Public Instruction. In fact, the person in charge 
of implementing the Texas system was a former legislator, which aided in the 
department's communication with the legislature. 

By comparison, the Ohio EMIS was led by a middle-level manager from within 
ODE whose existing duties were not reassigned when the EMIS became that individual's 
responsibility. 
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APPENDIX C 

HISTORY OF STATE AND SCHOOL DISTRICT DATA COLLECTION 



Prior to the 1979 development of the Ohio Education Computer Network (OECN), 
school districts were collecting budgeting, attendance, and other student-specific 
information for their own needs, while the state was collecting limited financial and 
other information about school districts. Although the data the state and districts were 
collecting had some overlap, the state's information was too inaccurate to make 
legitimate financial comparisons among school districts. 

During this time period, school districts' data collection systems were mostly 
manual and unique to each district Each school district defined the data they collected 
its own way; there was no standardization across districts. 

With the creation of the Uniform State Accounting System (USAS) in 1976, which 
established standard, but very broad revenue, cost, and expenditure categories among 
all school districts, ODE was able to make more legitimate financial comparisons among 
school districts. With the creation of the OECN in 1979, school districts began reporting 
USAS data to the state electronically. 

State policy makers created the EMIS because the USAS system could not answer 
detailed cost and other questions about individual students, staff, and school buildings. 
Although school districts were collecting information on individual students and staff 
as EMIS requires, it was not tied to costs and the state did not have access to much of 
the district information. 

The EMIS initiated more standardization of data. Financial, student, and staff 
data are now defined by the state, making comparisons among school districts more 
accurate. 
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